feat(nucleus): Add NucleusClient.merge_model_runs() - #474

Adds NucleusClient.merge_model_runs(model_run_ids, name=None, *, metadata=None), wrapping POST /v1/nucleus/modelRun/merge.

Requires scaleapi PR https://github.com/scaleapi/scaleapi/pull/156919, which adds the endpoint.

Why

A benchmark evaluation names a single model run, and a benchmark's items may span several datasets. A model whose predictions were uploaded as separate runs — one per dataset, or one per inference batch — has no single run covering the benchmark, so every uncovered item scores as a false negative. Merge first, wait for the copy to finish, then pass the new run to create_benchmark_evaluation_v2().

result = client.merge_model_runs(["run_abc", "run_def", "run_ghi"])
result["job"].sleep_until_complete()
evaluation = client.create_benchmark_evaluation_v2(
    benchmark_id, result["model_run_id"]
)

Semantics

Asynchronous. The endpoint returns 202 immediately with {"model_run_id", "dataset_ids", "job"}. The new run exists and is authorized right away, but its predictions are copied by a background job, so the run is empty until the job completes. Wait on result["job"] (an AsyncJob) before evaluating — otherwise the evaluation scores the not-yet-copied items as false negatives, the very failure this feature is meant to fix.

All source runs must belong to the same model; merging across models is rejected server-side (a run's model is its provenance, read by eval, leaderboards and the model page).

Full union — predictions are copied, never deduplicated, and the source runs are left untouched. If two source runs predict on the same item with the same annotation_id, the colliding id is rewritten rather than dropped. The copy's counts (predictions_copied, predictions_ignored, annotation_ids_rewritten) and any errors are reported on the job, not in the immediate response.

Verification

The test suite requires live API keys (conftest.py hard-asserts on NUCLEUS_PYTEST_API_KEY), so it could not be run locally, and there is no local backend to exercise the round trip against. Verified offline against the endpoint's Joi schema and handler in scaleapi #156919: the method builds the exact payload the server accepts (model_run_ids, optional name / metadata — no model_id, which the schema rejects and which the same-model constraint makes meaningless), posts to modelRun/merge, wraps the 202 response's job_id in an AsyncJob, and raises on fewer than two distinct run ids before making a request.

Note on the version bump

Bumped to 0.21.0. PR #473 also bumps pyproject.toml / CHANGELOG.md; whichever lands second will need a trivial rebase on those two files.

🤖 Generated with Claude Code

Greptile Summary

Adds NucleusClient.merge_model_runs() for asynchronously combining predictions from multiple same-model runs into a new run.

• Deduplicates source run IDs while preserving their order and validates that at least two distinct runs were supplied.
• Sends optional name and metadata fields to the merge endpoint.
• Returns the merged model-run ID, covered dataset IDs, and an AsyncJob handle for monitoring the background copy.
• Adds the new request-key constant and release notes.

sequenceDiagram
    participant U as SDK caller
    participant C as NucleusClient
    participant A as Nucleus API
    participant J as AsyncJob
    participant E as Evaluation V2
    U->>C: merge_model_runs(run IDs, name, metadata)
    C->>C: Deduplicate and validate run IDs
    C->>A: POST modelRun/merge
    A-->>C: model_run_id, dataset_ids, job metadata
    C-->>U: model_run_id, dataset_ids, AsyncJob
    U->>J: sleep_until_complete()
    J->>A: Poll job status
    A-->>J: Completed
    U->>E: create_benchmark_evaluation_v2(benchmark, merged run)

_{Reviews (6): Last reviewed commit: "Merge branch 'master' into lukeschaefer/..." | Re-trigger Greptile}